Big Data, Right Tools: Computational Resources for Empirical Research 2014

Transcription

1 Big Data, Right Tools: Computational Resources for Empirical Research 2014 Dokyun Lee, PhD Candidate, OPIM Dept. July 30, 2014 The aim of this course is to familiarize beginning Wharton PhD studentswithbothpubliclyavailable and Wharton specific cutting-edge tools and resources available to them for empirical research. The course is primarily concerned with acquiring, cleaning, storing, and managing data for empirical business research, and it will provide hands-on experience using variety of modern computing tools. Throughout the course, we will have toy example projects through which we will cover different aspects of data analysis tools and technologies. At the end of the course, students will know what tools are most appropriate for different data analysis jobs at hand. The following is a list of topics that I propose the course covers. Without extra topics, the topics outlined below would require approximately 8 sessions. The syllabus will be adjusted throughout the course to reflect class s interest and level. To borrow from Emerson s famous quote: Assertion of supremacy of one tool is a hobgoblin of little minds. different tools have comparative advantages whether it be a convenience-power tradeoff or support-power tradeoff or etc. Be open-minded and try different tools. Contents 1 Introduction: Syllabus & Installation of Required Software & Basic UNIX (1 lecture) Overview of syllabus topics Different tools are good for different jobs - example with statistical softwares (R, STATA, Matlab etc) Hierarchy of tools Gauge audience background Software Install & Setting up UNIX Wharton Grid account ( Hugh MacMullan -Director of Research Computing) Homeworks: Grid Computing (1 lecture - Hugh MacMullan - Director of Research Computing Capability, Usage, and Tips Speaker: Director of Wharton Research Computing (Hugh MacMullan) Intro to Programming Through Scripting Languages (0.5-1 lecture) Unix environment & Basic Procedural Programming Example: Python Basic data structures (quickly go through, details to be explored at home) Basic control flows (quickly go through, details to be explored at home) Homeworks: Acquiring and Cleaning Data Using Regular Expressions and APIs (2-2.5 lecture) Brief Overview of Wharton Data and resources (WRDS, WCAI, BakerCenter,Mackcenter etc) & Outside resources (Brief intro) Regular Expression

2 4.3 Basic Web Scraping Beyond Basic Web Scraping - Scrapy Framework Using APIs Homeworks: R (Guest Lecturing Ari Friedman lecture) Overview of R: What is it good for? what is it terrible for? Language architecture RMarkdown in Rstudio dplyr for data manipulation Big Data, Data Mining, and Causal Inference for Business Research (1 lecture) Predictive Analytics vs Causal Inference Big Data and Data Mining for Empirical Business Researchers Like Us Scratching the Surface of Natural Language Processing (1-2lectures) 6 1 Introduction: Syllabus & Installation of Required Software & Basic UNIX (1 lecture) 1.1 Overview of syllabus topics 1.2 Different tools are good for different jobs - example with statistical softwares (R, STATA, Matlab etc) Going through different tools that will be used in Stat/Econometric classes. Focused on comparing strength and weakness (e.g. speed, package availability). 1.3 Hierarchy of tools Learn hierarchyof tools. Low-Level -> Prototyping/Scripting -> Statistical Softwares. Just like bias-variance trade off in statistics or space-time tradeoff in computer algorithms, using different tools have different trade off wether it s simplicity-control tradeoff or simplicity-speed trade off. 1.4 Gauge audience background 1.5 Software Install & Setting up UNIX Wharton Grid account ( Hugh MacMullan- Director of Research Computing) Setting up environments and getting required softwares will be left to individuals for homeworks. Course will be taught with Unix-like environment (Mac OS). Window users can log into Wharton Grid computing network and work from there. Apply for your account here Homeworks: 1. Get unix environment set up & learn some commands (a) read more about unix environment here (b) At minimum, do tutorial 1. Others if interested. 2. Install Python & Choose your favorite text editor (a) Download Enthought s pre-packaged python version (EDP or Canopy) (b) It s free for edu s (c) (d) For editing in wharton unix grid, do look at editors such as emacs,viorpico. 2

3 2 Grid Computing (1 lecture - Hugh MacMullan - Director of Research Computing 2.1 Capability, Usage, and Tips 2.2 Speaker: Director of Wharton Research Computing (Hugh MacMullan) 3 Intro to Programming Through Scripting Languages (0.5-1 lecture) 3.1 Unix environment & Basic Procedural Programming Learning basic unix usage and writing a simple helloworld.py pythonprogram. Pythonwillbeusedthroughout the course. 3.2 Example: Python We ll cover Python since it is easy to learn, can be very fast, has great community online, and has great scientific computing packages in addition to data mining and text mining packages. 3.3 Basic data structures (quickly go through, details to be explored at home) 3.4 Basic control flows (quickly go through, details to be explored at home) if for while logical statements basic I/O 3.5 Homeworks: 1. Try out writing some simple python code and experiment with alldatastructuresandloops 2. Get comfortable with IDLE and writing a simple python code. IDLEX is a better version if you can install it. 4 Acquiring and Cleaning Data Using Regular Expressions and APIs (2-2.5 lecture) 4.1 Brief Overview of Wharton Data and resources (WRDS, WCAI, Baker Center, Mack center etc) & Outside resources (Brief intro) Where can one obtain data? Data hunting and resources. 3

4 4.2 Regular Expression I spent X hours manually cleaning the data. I hated it <- say no more. Regular Expression is a tool for searching, and processing text data that will make data cleaning much faster. Regular expression is a set of rules for representing text patterns and it s not so hard yet takes some practice. You will need to have an online cheatsheet always open for a while. 4

5 We will cover regex in python. It is true that perl popularized regex but anything you can do in perl regex, you can do in python regex. And python has slew of benefits over perl - perl used to be cool in the 90s and early 2000s. 4.3 Basic Web Scraping HTML GET vs POST Toy examples 4.4 Beyond Basic Web Scraping - Scrapy Framework SCRAPY < is a great tool for scraping web that s written in python. This is a framework that will take some time to learn but once familiarized, it can scrape basic websites in less than an hour. We will not go into more advanced scraping that require headless browser to bypass AJAX calls. 4.5 Using APIs There are tons of data online. Some providers have their own APIs (Application Programming Interface) ready to use for easy access to the data. When they don t, one must scrape the data. Check out this website < for getting resources on APIs available online. The website catalogs and provide search engine for freely available APIs online. Price related APIs: < Examples (Real estate data or Amazon Product reviews) Redfin.com < WalkScore < Amazon APIs < 4.6 Homeworks: 1. For Regular expression (a) Practice REGEX! 2. For APIs (a) Choose your favorite APIs. Come prepared to share the API with the class. What does it offer and why do you like it? (b) Obtain necessary access then write a simple call to get some data 3. Create and upload the data obtained in API lecture. 5 R (Guest Lecturing Ari Friedman lecture) Some of you have used R in the past, some have not. Either way, I believe this will be helpful. Ari is an R expert. 5

6 5.1 Overview of R: What is it good for? what is it terrible for? Language architecture 5.2 RMarkdown in Rstudio 5.3 dplyr for data manipulation 6 Big Data, Data Mining, and Causal Inference for Business Research (1 lecture) 6.1 Predictive Analytics vs Causal Inference 6.2 Big Data and Data Mining for Empirical Business Researchers Like Us 7 Scratching the Surface of Natural Language Processing (1-2lectures) Bird-eye view of NLP will be presented. Theory will not be taught due to time contraint. What will be covered is some basic idea of what can be done, how to do it, and what packages to use. 6