Experimental Finance, IEOR. Mike Lipkin, Alexander Stanton

Transcription

1 Experimental Finance, IEOR Mike Lipkin, Alexander Stanton

2 Housekeeping Make sure your tables/views/functions are prefixed with UNI ID One zip file please, containing other files Describe your process Project groups: Everyone settled? Experimental Finance Mike Lipkin, Alexander Stanton Page 2

3 Outline Some SQL queries you can help me with Creating tables and temporary tables Creating Indices Experimental Finance Mike Lipkin, Alexander Stanton Page 3

4 Strike calculation SELECT strike/1000 FROM option_price_view Strike Correct: SELECT CAST(strike AS FLOAT)/ FROM option_price_view Use the function xf.dbo.formatstrike()!! Experimental Finance Mike Lipkin, Alexander Stanton Page 4

5 Count number of options by week for a month SELECT DATEPART(week,date), COUNT(DISTINCT bestbid) FROM option_price_2000_02 GROUP BY DATEPART(week,date) Experimental Finance Mike Lipkin, Alexander Stanton Page 5

6 Count number of options by week for a month SELECT DATEPART(week,date), COUNT(DISTINCT bestbid) FROM option_price_2000_02 GROUP BY DATEPART(week,date) counting only distinct best bids probably not useful count(*) for shorthand makes it impossible to mistake DISTINCT SELECT DATEPART(week,date), COUNT(*) FROM option_price_2000_02 GROUP BY DATEPART(week,date) Week Count Experimental Finance Mike Lipkin, Alexander Stanton Page 6

7 Count number of options by week for a month SELECT DATEPART(week,date), COUNT(DISTINCT bestbid) FROM option_price_2000_02 GROUP BY DATEPART(week,date) counting only distinct best bids probably not useful count(*) for shorthand makes it impossible to mistake DISTINCT SELECT DATEPART(week,date), COUNT(*) FROM option_price_2000_02 GROUP BY DATEPART(week,date) First and last week are hardly ever complete This Matters (and not many pointed this out) Week Count Experimental Finance Mike Lipkin, Alexander Stanton Page 7

8 Query Comparison SELECT ticker FROM security s LEFT JOIN security_price sp ON s.securityid=sp.securityid AND sp.date=' WHERE sp.closeprice is null SELECT ticker FROM security s LEFT JOIN security_price sp ON s.securityid=sp.securityid WHERE sp.closeprice is null AND sp.date=' ' Many assertions and theories that they do one thing or the other TEST IT! Experimental Finance Mike Lipkin, Alexander Stanton Page 8

9 How many unique tickers traded on a given day? SELECT count(distinct ticker) SELECT count(distinct ticker) FROM security s FROM security s LEFT JOIN security_price sp LEFT JOIN security_price sp ON s.securityid=sp.securityid ON s.securityid=sp.securityid WHERE sp.date = ' AND sp.date = WHERE sp.securityid is not null SELECT count(distinct ticker) SELECT count(distinct ticker) FROM security s FROM security s LEFT JOIN security_price sp WHERE s.ticker NOT IN ON s.securityid=sp.securityid (SELECT s1.ticker FROM security s1 AND sp.date = ' LEFT JOIN security_price sp ON s1.securityid=sp.securityid AND sp.date= WHERE sp.securityid is null) Experimental Finance Mike Lipkin, Alexander Stanton Page 9

10 How many unique tickers traded on a given day? SELECT count(distinct ticker) SELECT count(distinct ticker) FROM security s FROM security s LEFT JOIN security_price sp LEFT JOIN security_price sp ON s.securityid=sp.securityid ON s.securityid=sp.securityid WHERE sp.date = ' AND sp.date = WHERE sp.securityid is not null SELECT count(distinct ticker) SELECT count(distinct ticker) FROM security s FROM security s LEFT JOIN security_price sp WHERE s.ticker NOT IN ON s.securityid=sp.securityid (SELECT s1.ticker FROM security s1 AND sp.date = ' LEFT JOIN security_price sp ON s1.securityid=sp.securityid AND sp.date= WHERE sp.securityid is null) Experimental Finance Mike Lipkin, Alexander Stanton Page 10

11 The right way SELECT count(distinct ticker) FROM security s INNER JOIN security_price sp ON s.securityid=sp.securityid WHERE sp.date =' Only use OUTER JOINS when: Data is required from a table even when there is none in the joining table Using a WHERE clause that satisfies a NULL condition (Set theory: relative compliment), e.g.: SELECT count(distinct ticker) FROM security s LEFT OUTER JOIN security_price sp ON s.securityid=sp.securityid WHERE sp.date is null Experimental Finance Mike Lipkin, Alexander Stanton Page 11

12 SQL Performance Are they the same? Is one faster than the other? SELECT TOP 1 opv1.strike, opv2.strike FROM option_price_view opv1 INNER JOIN option_price_view opv2 ON opv2.securityid=opv1.securityid AND opv2.date=opv1.date AND opv2.expiration=opv1.expiration AND opv2.strike<opv1.strike ORDER BY opv1.strike DESC SELECT TOP 1 opv1.strike, opv1.strike FROM option_price_view opv1 WHERE opv1.strike<(select top 1 strike from option_price_view opv2 WHERE opv2.securityid=opv1.securityid AND opv2.date=opv1.date AND opv2.expiration=opv1.expiration AND opv2.strike=opv1.strike ORDER BY strike DESC) ORDER BY opv1.strike DESC Experimental Finance Mike Lipkin, Alexander Stanton Page 12

13 SQL Performance Are they the same? Is one faster than the other? SELECT TOP 1 opv1.strike, opv2.strike FROM option_price_view opv1 INNER JOIN option_price_view opv2 ON watch ordering! opv2.securityid=opv1.securityid AND opv2.date=opv1.date AND opv2.expiration=opv1.expiration AND opv2.strike<opv1.strike ORDER BY opv1.strike DESC, opv2.strike DESC 20 sec. SELECT TOP 1 opv1.strike, opv1.strike FROM option_price_view opv1 WHERE opv1.strike<(select top 1 strike from option_price_view opv2 WHERE opv2.securityid=opv1.securityid AND opv2.date=opv1.date AND opv2.expiration=opv1.expiration AND opv2.strike=opv1.strike ORDER BY strike DESC) ORDER BY opv1.strike DESC, opv2.strike DESC 4+ hours Experimental Finance Mike Lipkin, Alexander Stanton Page 13

14 SQL Performance Better yet: for readability and cross-checking: SELECT strike =dbo.formatstrike(opv1.strike), strikebelow =dbo.formatstrike(opv2.strike), strikeabove =dbo.formatstrike(opv3.strike) FROM option_price_2005_01 opv1 INNER JOIN option_price_2005_01 opv2 ON opv2.securityid=opv1.securityid AND opv2.date=opv1.date AND opv2.expiration=opv1.expiration AND opv2.strike<opv1.strike INNER JOIN option_price_2005_01 opv3 ON opv3.securityid=opv1.securityid AND opv3.date=opv1.date AND opv3.expiration=opv1.expiration AND opv3.strike>opv1.strike Strike Below Above Experimental Finance Mike Lipkin, Alexander Stanton Page 14

15 SQL Performance Better yet: for readability and cross-checking: SELECT strike =dbo.formatstrike(opv1.strike), strikebelow =dbo.formatstrike(opv2.strike), strikeabove =dbo.formatstrike(opv3.strike) FROM option_price_2005_01 opv1 INNER JOIN option_price_2005_01 opv2 ON opv2.securityid=opv1.securityid AND opv2.date=opv1.date AND opv2.expiration=opv1.expiration AND opv2.strike<opv1.strike INNER JOIN option_price_2005_01 opv3 ON opv3.securityid=opv1.securityid AND opv3.date=opv1.date AND opv3.expiration=opv1.expiration AND opv3.strike>opv1.strike WHAT IS WRONG HERE? Strike Below Above Experimental Finance Mike Lipkin, Alexander Stanton Page 15

16 SQL Performance CORRECTED: SELECT strike =dbo.formatstrike(opv1.strike), strikebelow =dbo.formatstrike(max(opv2.strike)), strikeabove =dbo.formatstrike(min(opv3.strike)) FROM option_price_2005_01 opv1 INNER JOIN option_price_2005_01 opv2 ON opv2.securityid=opv1.securityid AND opv2.date=opv1.date AND opv2.expiration=opv1.expiration AND opv2.strike<opv1.strike INNER JOIN option_price_2005_01 opv3 ON opv3.securityid=opv1.securityid AND opv3.date=opv1.date AND opv3.expiration=opv1.expiration AND opv3.strike>opv1.strike GROUP BY opv1.strike Strike Below Above Experimental Finance Mike Lipkin, Alexander Stanton Page 16

17 SQL Performance: WHERE clauses SELECT count(distinct ticker) FROM security s INNER JOIN security_price sp ON s.securityid=sp.securityid WHERE sp.date =' SELECT count(distinct ticker) FROM security s INNER JOIN security_price sp ON s.securityid=sp.securityid WHERE sp.date IS NOT NULL Experimental Finance Mike Lipkin, Alexander Stanton Page 17

18 SQL Performance: WHERE clauses SELECT count(distinct ticker) FROM security s INNER JOIN security_price sp ON s.securityid=sp.securityid WHERE sp.date =' seconds to find 8240 rows SELECT count(distinct ticker) FROM security s INNER JOIN security_price sp ON s.securityid=sp.securityid WHERE sp.date IS NOT NULL 17 seconds to go through 33 million rows Beware of functions in where clauses e.g. WHERE f(col1)>x - more on this later. Experimental Finance Mike Lipkin, Alexander Stanton Page 18

19 How many stocks trade at the beginning of the year? int = 1996 While(@Year < 2011) BEGIN as 'Year, COUNT(distinct SecurityID) as 'Securities FROM dbo.security_price WHERE Date = (select Top 1 Date FROM dbo.security_price WHERE YEAR(date) +1 END Experimental Finance Mike Lipkin, Alexander Stanton Page 19 Year Securities

20 How many stocks trade at the beginning of the year? int = 1996 While(@Year < 2011) BEGIN as 'Year, COUNT(distinct SecurityID) as 'Securities FROM dbo.security_price WHERE Date = (select Top 1 Date FROM dbo.security_price WHERE YEAR(date) +1 END 30 Queries + hardcoding values, never good Experimental Finance Mike Lipkin, Alexander Stanton Page 20 Year Securities

21 How many stocks trade at the beginning of the year? SELECT "stocks"=count(distinct securityid),date FROM SECURITY_PRICE s WHERE date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' and ' ') [..] GROUP BY date ORDER BY date 16 Queries, a little better Experimental Finance Mike Lipkin, Alexander Stanton Page 21

22 How many stocks trade at the beginning of the year? SELECT "stocks"=count(distinct securityid),date FROM SECURITY_PRICE s WHERE date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' and ' ') or date= (select Top 1(date) from Security_Price where date between ' and ' ') [..] GROUP BY date ORDER BY date top 1 date between 1/1 and 1/5 is not precise. Takes +/- 4 minutes Experimental Finance Mike Lipkin, Alexander Stanton Page 22

23 How many stocks trade at the beginning of the year? SELECT COUNT(DISTINCT s.ticker) FROM [..] WHERE sn1.date = (SELECT MIN(date) FROM [..] WHERE MONTH(date)=1 and DAY(date)<=3) Experimental Finance Mike Lipkin, Alexander Stanton Page 23

24 How many stocks trade at the beginning of the year? SELECT COUNT(DISTINCT s.ticker) FROM [..] WHERE sn1.date = (SELECT MIN(date) FROM [..] WHERE MONTH(date)=1 and DAY(date)<=3) - Friday New Years day causes missing data - Do not necessarily restrict your data to speed up queries. Without the DAY() restriction it would not be much slower. - Look for several orders of magnitude when optimizing queries Experimental Finance Mike Lipkin, Alexander Stanton Page 24

25 How many stocks trade at the beginning of the year? SELECT FROM security_price WHERE month(date)=1 GROUP BY year(date) ORDER BY year(date) min(date) AS FirstDayOfYear, count(securityid) AS NumOfStocksTrading Year Count Experimental Finance Mike Lipkin, Alexander Stanton Page 25

26 How many stocks trade at the beginning of the year? SELECT FROM security_price WHERE month(date)=1 GROUP BY year(date) ORDER BY year(date) min(date) AS FirstDayOfYear, count(distinct securityid) AS NumOfStocksTrading Need to count distinct securities what else is wrong? Year Count Experimental Finance Mike Lipkin, Alexander Stanton Page 26

27 How many stocks trade at the beginning of the year? SELECT FROM security_price WHERE month(date)=1 GROUP BY year(date) ORDER BY year(date) min(date) AS FirstDayOfYear, count(distinct securityid) AS NumOfStocksTrading Year Count Need to count distinct securities This counts all tickers that traded in the first month of each year. Not the first day of the year. Instead: SELECT YEAR(date), COUNT(DISTINCT securityid) FROM security_price WHERE date IN (SELECT min(date) [from SP] GROUP BY YEAR(date)) GROUP BY YEAR(date) Experimental Finance Mike Lipkin, Alexander Stanton Page 27

28 Close Prices SELECT date, ticker, closeprice FROM security s INNER JOIN security_price sp ON s.securityid=sp.securityid WHERE s.ticker = 'C AND year(sp.date) = '1999 Experimental Finance Mike Lipkin, Alexander Stanton Page 28

29 Close Prices SELECT date, ticker, closeprice FROM security s INNER JOIN security_price sp ON s.securityid=sp.securityid WHERE s.ticker = 'C AND year(sp.date) = ' Experimental Finance Mike Lipkin, Alexander Stanton Page 29

30 Adjustment Factor Exists on security_price, option_price SELECT closeprice*adjustmentfactor for consistent data SELECT closeprice*adjf/max(adjf) in a date range yields prices adjusted to the last date in the queried data set Experimental Finance Mike Lipkin, Alexander Stanton Page 30

31 At The Moneys NOT GOOD: SELECT s1.date,o.strike,s1.closeprice FROM security s INNER JOIN security_price s1 ON s.securityid=s1.securityid INNER JOIN option_price_2005_03 o ON s.securityid=o.securityid AND s1.date=o.date WHERE s.ticker='msft' AND ABS(s1.closeprice-dbo.formatStrike(o.strike))<=2.5 ORDER BY s1.date GOOD: SELECT [..] WHERE s.ticker='msft' AND ABS(S-K) = (SELECT MIN(ABS(S-K)) FROM [..] WHERE ID=ID and Date=Date) ORDER BY s1.date ATM: by now you know which one to use. If not, ask Mike during office hours or you will be sorry. Experimental Finance Mike Lipkin, Alexander Stanton Page 31

32 Summary Think about boundary conditions Validate and test start building your portfolio of checks and exclusions: -99 deltas Volume>0 Negative prices Implied Vols breaking down near expiration Interest rates Date ranges Once you understand and solve a data issue, package it and make sure you reuse it. Experimental Finance Mike Lipkin, Alexander Stanton Page 32

33 Summary Creating Tables and Indices Experimental Finance Mike Lipkin, Alexander Stanton Page 33

34 Introduction Create tables to: Store new information De-normalize tables Solve problems that cannot be addressed in a single query Speed up access to data that takes a long time to compute Store historical data sets for comparison Always prefix objects with your UNI ID when creating tables, functions, stored procedures Use proper naming conventions, not temp1 through temp68 In the lab, do not create tables with very large datasets!! Experimental Finance Mike Lipkin, Alexander Stanton Page 34

35 CREATE TABLE Syntax CREATE TABLE mytable ( ) mytableid INT NOT NULL, ticker VARCHAR(7) NOT NULL, description VARCHAR(255) NULL, type CHAR(2), istraded BIT NOT NULL, PRIMARY KEY(myTableID) Types: - INT, TINYINT, SMALLINT, BIGINT, DECIMAL, NUMERIC, FLOAT, REAL - MONEY - VARCHAR - BIT - DATETIME - TIMESTAMP (auto updates when a row is inserted/updated) - TEXT/BINARY/IMAGE Experimental Finance Mike Lipkin, Alexander Stanton Page 35

36 CHAR vs. VARCHAR CHAR: Fixed length, whether there is data or not Ticker CHAR(10) containing MSFT stored as MSFT Storage implications Microsoft SQL accepts WHERE ticker= MSFT - MANY DO NOT VARCHAR(size): variable length and stored as a byte array the size of the actual value Size specifies maximum size constraint unpredictable table size Slightly slower page retrieval Experimental Finance Mike Lipkin, Alexander Stanton Page 36

37 NUMERIC TYPES DECIMAL FLOAT REAL MONEY INT, BIGINT, TINYINT Be careful with exact and approximate numeric types Also: rounding errors moving from sql -> Excel, Matlab, C++ Experimental Finance Mike Lipkin, Alexander Stanton Page 37

38 Primary Keys Unique per row Cannot be NULL Can consist of one or more columns, e.g: PRIMARY KEY (securityid, date) Surrogate keys for Primary Keys Used for convenience Key does not represent inherent information Need a way to define the surrogate key: definition: myid INT NOT NULL IDENTITY(seed,increment) PRIMARY KEY e.g. myid INT NOT NULL IDENTITY(1,1) PRIMARY KEY Experimental Finance Mike Lipkin, Alexander Stanton Page 38

39 Inserting into tables INSERT INTO mytable (date, ticker, price) VALUES ( , MSFT, 20.5) SELECT securityid, date, ticker, closeprice INTO mytable FROM security INNER JOIN security_price ON [ ] BULK INSERT mytable FROM c:\data.txt Command for fast loading of data from text files Text File must reside on the server Can cause full table lock / no constraint checks IVY is set up this way automatically creates table with the correct columns and column types Experimental Finance Mike Lipkin, Alexander Stanton Page 39

40 Temporary Tables Temporary tables are fast, and very useful A table name starting with # implies a temporary table Only exist for the duration of the connection Stored in RAM unless size limitations force caching to disk Good idea to de-allocate the table using DROP #mytable when finished Can get corrupted - not good for anything beyond temporary storage CREATE TABLE #mytable(col1,col2, ) OR SELECT securityid, date, ticker, closeprice INTO #myothertable FROM security INNER JOIN security_price ON [ ] Experimental Finance Mike Lipkin, Alexander Stanton Page 40

41 INDEX Syntax CREATE UNIQUE CLUSTERED INDEX myindexname ON security_price (date, volume, ) There can only be one and only one CLUSTERED Index. The Directive tells the database to physically rearrange and store the data according to the index. Which of the queries below take advantage of the index? SELECT securityid from [SP] WHERE date>[x] AND volume>20,000,000 SELECT securityid from [SP] WHERE date>[y] AND volume>1,000 Indices take up space, potentially duplicating the storage used by the column(s) being indexed if all columns are included VIEWS can also be indexed Experimental Finance Mike Lipkin, Alexander Stanton Page 41

42 INDEX Examples CREATE UNIQUE CLUSTERED INDEX i1 ON option_price (date, security_id, strike, expiration, delta) CREATE UNIQUE INDEX i2 ON option_price (security_id, date, expiration, delta) SELECT * FROM option_price WHERE WHERE date > WHERE securityid = [MSFT] WHERE strike=25000 WHERE strike=25000 AND securityid=[msft] WHERE delta>40 AND delta<50 AND date> Experimental Finance Mike Lipkin, Alexander Stanton Page 42

43 Indices w.r.t. Functions If functions are used in the where clause, indices using that column cannot be used! (unless the index is relevant w.r.t. other columns.) SELECT * FROM security_price sp WHERE datepart(year, sp.date)=2004 (Much) better: SELECT * FROM security_price sp WHERE sp.date>= and sp.date< OR SELECT * FROM security_price sp WHERE DATEDIFF(dd,date,GETDATE())<=10 (Much) better: SELECT * FROM security_price sp WHERE sp.date>=dateadd(dd,-10,getdate()) Experimental Finance Mike Lipkin, Alexander Stanton Page 43

44 Indices Note: Be careful with dates WHERE sp.date>= AND sp.date< Expands to: WHERE sp.date>= :00:00 AND sp.date< :00:00 Experimental Finance Mike Lipkin, Alexander Stanton Page 44

45 Physical Data Retrieval, Covering Indices How the database retrieves data: Request is made for table data Index rapidly finds the physical row locator Loads the relevant page from disk (all columns) Appropriate columns are returned to the SELECT statement Covering indices: Have all data required to fulfill the SELECT statement. e.g.: CREATE UNIQUE INDEX myindexname ON option_price (securityid, date, expiration, strike) No second physical data read required Much faster, but of course query dependent Indices are very powerful and worth learning well Experimental Finance Mike Lipkin, Alexander Stanton Page 45

46 More fun stuff What is the best way to get a list of trade dates? When do options expire? When are Earnings? Deals and special settlements Experimental Finance Mike Lipkin, Alexander Stanton Page 46

47 More fun stuff What is the best way to get a list of trade dates? SELECT DISTINCT date FROM security_price When do options expire? When are Earnings? Deals and special settlements Experimental Finance Mike Lipkin, Alexander Stanton Page 47

48 More fun stuff What is the best way to get a list of trade dates? SELECT DISTINCT date FROM security_price When do options expire? Generally Friday Night at 5:30pm, third Friday of the month Index options (AM/PM) Good Friday they expire Thursday SELECT DISTINCT expiration FROM option_price_view When are Earnings? Deals and special settlements Experimental Finance Mike Lipkin, Alexander Stanton Page 48

49 More fun stuff What is the best way to get a list of trade dates? SELECT DISTINCT date FROM security_price When do options expire? Generally Friday Night at 5:30pm, third Friday of the month Index options (AM/PM) Good Friday they expire Thursday SELECT DISTINCT expiration FROM option_price_view When are Earnings? Pre-open Post close Intraday (generally trading is halted if it s a leaked announcement) Deals and special settlements Experimental Finance Mike Lipkin, Alexander Stanton Page 49

50 More fun stuff What is the best way to get a list of trade dates? SELECT DISTINCT date FROM security_price When do options expire? Generally Friday Night at 5:30pm, third Friday of the month Index options (AM/PM) Good Friday they expire Thursday SELECT DISTINCT expiration FROM option_price_view When are Earnings? Pre-open Post close Intraday (generally trading is halted if it s a leaked announcement) Deals and special settlements Options may pay out different number of shares of stock (See front month and leaps in APOL in DEC 2005) Experimental Finance Mike Lipkin, Alexander Stanton Page 50

51 Adjustment Factors For the problem sets, check IVYs reference for adjustment factors on the security_price, option_price and dividend tables. There are several different adjustment factors and they are used in different scenarios you need to understand them! We ll be covering these in more details next time Experimental Finance Mike Lipkin, Alexander Stanton Page 51

52 Coming Up Last important building block: Functions and Stored Procedures Experimental Finance Mike Lipkin, Alexander Stanton Page 52