Case study: accessing financial data

Transcription

1 Case study: accessing financial data Prof. Mauro Gaspari:

2 Methods for accessing databases What methods exist to access financial databases? Basically there are several approaches to the use of databases:

3 Local versus Remote Local access: downloading a copy of the database or files of data (e.g. by FTP) and direct access to the database on your PC. On line access using a remote database server.

4 User Interface Versus Programs Using an user interface (local or remote): for example a Web page form (which interfaces to a search program on the database server). For instance the Wharton Research Data Services WRDS ( Using a program (e.g a Python script) on your PC: pyodbc (Open Database Connectivity) supporting local and remote access to the database (using SQL). Web services (web based function calls): xignite ( yahoo finance ( Google finance (

5 Web Versus Web Services Ordinary Web HTTP Request Client PC Browser HTTP Response HTML or Other Document Webserver Application Webserver Ordinary web was created to download documents

6 Web Services HTTP Request Client PC Python Program HTTP Response Service Object Webserver Message Web services, in contrast to web, provide program-to-program communication. Programs on the webserver are called service objects. Messages follow ad-hoc formats (HTTP, XML SOAP).

7 Web Services Parameters Client PC Python program Result of Calculation Service Object Webserver Message The sending program sends parameters for the calculation. The service object does the calculations and sends back results.

8 Get Stock Daily Data Pandas Datareader Yahoo finance Quandl IEX Finance

9 Pandas Dataframe All these modules return data in Pandas dataframe. Pandas is Python module providing high-performance, easy-to-use data structures and data analysis tools. The dataframe is the primary pandas data structure: Two-dimensional tabular data structure with labeled axes (rows and columns). Basically its is a dict-like container for Series objects. Arithmetic operations align on both row and column labels. Can be extended.

10 Dataframe objects options class pandas.dataframe(data,index,columns,dtype,copy) data dict, lists, or other Index Index or array-like: Index to use for resulting frame. default np.arange(n) columns -> Index or array-like: Column labels to use for resulting frame. default np.arange(n) dtype dtype, default None Copy boolean, default False

11 Constructing DataFrame from a dictionary >>> d = {'col1': [1, 2], 'col2': [3, 4]} >>> df = pd.dataframe(data=d) >>> df col1 col >>> df.dtypes col1 int64 col2 int64 dtype: object >>> df['col1'][0] 1 >>> df['col2'][1] 4

12 Constructing Dataframe from a List >>> df2 = pd.dataframe([1,2,3,4]) >>> df >>> df3 = pd.dataframe([[1,2,3,4],[5,6,7,8]]) >>> df >>> df3[0][0] 1 >>> df3[3][0] 4

13 Constructing DataFrame from numpy ndarray: >>> df2 = pd.dataframe(np.random.randint(low=0, high=10, size=(5, 5)),... columns=['a', 'b', 'c', 'd', 'e']) >>> df2 a b c d e

14 Pandas Datareader from pandas_datareader import data as pdr # download dataframe data = pdr.get_data_yahoo("spy", start=" ", end=" ") # download Panel data = pdr.get_data_yahoo(["spy", "IWM"], start=" ", end=" ") Yahoo!, Google Options, Google Quotes and EDGAR have been immediately deprecated

15 fix_yahoo_finance In 2017 Yahoo! finance has decommissioned their historical data API. fix-yahoo-finance is a temporary fix by scraping the data from Yahoo! finance using and return a Pandas DataFrame Extends pandas datareader

16 Example from pandas_datareader import data as pdr import fix_yahoo_finance as yf yf.pdr_override() # use fix_yahoo_finance # download dataframe data = pdr.get_data_yahoo("spy", start=" ", end=" ") # download Panel data = pdr.get_data_yahoo(["spy", "IWM"], start=" ", end=" ")

17 Main options tickers = ["SPY", "IWM", "..."] start = " " YYYY-MM-DD / datetime.datetime object (optional, defaults is ) End = " " (optional, defaults is Today_ group_by = 'ticker' group by ticker to access via data['spy'] (optional, default is 'column') auto_adjust = True adjust all OHLC automatically (optional, default is False) actions = True download dividend + stock splits data (optional, default is None) options are: True (returns history + actions); 'Open'; High'; 'Low' ;'Close' ;'Adj Close' ;'Volume'.

18 As a standalone module It can be used whitout pandas_datareader. The same options are supported. import fix_yahoo_finance as yf data = yf.download("spy", start=" ", end=" ")

19 Quandl Quandl delivers finactial and economic data from differen sources. Data in Quandl databases are constantly updated. Includes Stock, Future, Commodity, Currency, Interest Rate, Option, Fund, Index, Industry, and Economic Data A Web interface allows users to browse databases: Quandl provides and API that can be accessed directly from Python (support is also provided for R and excel). Datasets are formatted and ready for end-use, with time-series, data and embeddable automatic line charts. Supported formats: csv, json, excel, xml.

20 Getting Started You can download the Quandl Python module from PyPI or from GitHub. Follow the installation instructions here: Quandl is free but you must have a Quandl API key in order to download data. Set up your API key here: How to use: import quandl quandl.apiconfig.api_key = YOURKEY

21 Make time-series calls # gets the WTI Crude Oil Price from the US Department of Energy data = quandl.get("eia/pet_rwtc_d") # To set start and end dates: data = quandl.get("fred/gdp", start_date=" ",end_date=" ") # To request specific columns: data = quandl.get(["nse/oil.1", "WIKI/AAPL.4"]) # To request the last 5 rows: data = quandl.get("wiki/aapl", rows=5) # To change the sampling frequency: data = quandl.get("eia/pet_rwtc_d", collapse="monthly") # To perform elementary calculations on the data: data = quandl.get("fred/gdp", transformation="rdiff") # Data series parameters are summarized in this site: #

22 Tabular data Considering the Mergent Global Fundamentals dataset, specifically the MER/F1 table. This table is filterable on multiple columns, including compnumber, mapcode and reportdate writing request to rows with specific values for these (and all available) filters. The tables API is limited to 10,000 rows per call. However, when using the Python library, appending the argument paginate=true will bypass this limit. We recommend using paginate=true for all tabular calls. Some datasets can return more data than Python allows. If this occurs, you will need to further filter your call to download less data.

23 Examples of tabular calls # Download a Table data = quandl.get_table('mer/f1', paginate=true) # Column selection: data for Nokia (compnumber=39102) data = quandl.get_table('mer/f1', compnumber= 39102, paginate=true) # Column selection: Nokia (compnumber=39102) and Deutsche Bank (compnumber=2438) data = quandl.get_table('mer/f1', compnumber=[ 39102, 2438 ], paginate=true) # Closing prices: Apple (AAPL) and Microsoft (MSFT) from to data=quandl.get_table('wiki/prices',qopts={'columns':['ticker','date', 'close']}, ticker=['aapl', 'MSFT'], date={'gte':' ','lte': ' '})

24 IEX Finance from iexfinance import get_historical_data from datetime import datetime start_date=' ' end_date=' ' start_date = pd.to_datetime(start_date) end_date = pd.to_datetime(end_date) data = get_historical_data( GOOG, start=start_date, end=end_date, output_format='pandas')

25 Basic Web Scraping How to extract information from Web Sites. General method for transforming unstructured data (HTML) into databases or spreadshits. Python modules: Urllib2: can be used for fetching URLs. It defines functions and classes to help with URL actions. BeautifulSoup: It is an incredible tool for pulling out information from a webpage. You can use it to extract tables, lists, paragraph and you can also put filters to extract information from web pages.

26 Example >>> import urllib2 >>> from bs4 import BeautifulSoup >>> site = " 2018/392829/orariolezioni" >>> page = urllib2.urlopen(site) >>> soup = BeautifulSoup(page) >>> soup.title <title>orario delle lezioni di COMPUTER PROGRAMMING 2018/2019 \u2014 Universit\xe0 di Bologna</title> >>> soup.title.string u'orario delle lezioni di COMPUTER PROGRAMMING 2018/2019 \u2014 Universit\xe0 di Bologna' >>> all_tables=soup.find_all('table')