Science Cookbook. Practical Data. open source community experience distilled. Benjamin Bengfort. science projects in R and Python.

Transcription

1 Practical Data Science Cookbook 89 hands-on recipes to help you complete real-world data science projects in R and Python Tony Ojeda Sean Patrick Murphy Benjamin Bengfort Abhijit Dasgupta PUBLISHING open source community experience distilled BIRMINGHAM MUMBAI -

2 Preface 1 Chapter 1: Preparing Your Data Science Environment 7 Introduction 7 Understanding the data science pipeline 9 Installing R on Windows, Mac OS X, and Linux 11 Installing libraries in R and RStudio 14 Installing Python on Linux and Mac OS X 17 Installing Python on Windows 18 Installing the Python data stack on Mac OS X and Linux 21 Installing extra Python packages 24 Installing and using virtualenv 26 Chapter 2: Driving Visual Analysis with Automobile Data (R) 31 Introduction 31 Acquiring automobile fuel efficiency data 32 Preparing R for your first project 34 Importing automobile fuel efficiency data into R 35 Exploring and describing fuel efficiency data 38 Analyzing automobile fuel efficiency over time 43 Investigating the makes and models of automobiles 54 Chapter 3: Simulating American Football Data (R) 59 Introduction 59 Acquiring and cleaning football data 61 Analyzing and understanding football data 65 Constructing Indexes to measure offensive and defensive strength 74 Simulating a single game with outcomes decided by calculations 77 Simulating multiple games with outcomes decided by calculations 81

3 Chapter 4: Modeling Stock Market Data (R) 89 Introduction 89 Acquiring stock market data 91 Summarizing the data 93 Cleaning and exploring the data 96 Generating relative valuations 103 Screening stocks and analyzing historical prices 109 Chapter 5: Visually Exploring Employment Data (R) 117 Introduction 118 Preparing for analysis 119 Importing employment data into R 121 Exploring the employment data 123 Obtaining and merging additional data 125 Adding geographical information 129 Extracting state- and county-level wage and employment information 133 Visualizing geographical distributions of pay 136 Exploring where the jobs are, by industry 140 Animating maps for a geospatial time series 143 Benchmarking performance for some common tasks 149 Chapter 6: Creating Application-oriented Analyses Using Tax Data (Python) 153 Introduction 153 Preparing for the analysis of top incomes 155 Importing and exploring the world's top incomes dataset 156 Analyzing and visualizing the top income data of the US 165 Furthering the analysis of the top income groups of the US 174 Reporting with Jinja2 179 Chapter 7: Driving Visual Analyses with Automobile Data (Python) 187 Introduction 187 Getting started with I Python 188 Exploring I Python Notebook 191 Preparing to analyze automobile fuel efficiencies 196 Exploring and describing fuel efficiency data with Python 199 Analyzing automobile fuel efficiency over time with Python 202 Investigating the makes and models of automobiles with Python 211 Chapter 8: Working with Social Graphs (Python) 217 Introduction 217 Preparing to work with social networks in Python 220 Importing networks 222 HjD

4 Exploring subgraphs within a heroic network 225 Finding strong ties 230 Finding key players 234 Exploring the characteristics of entire networks 246 Clustering and community detection in social networks 248 Visualizing graphs 254 Chapter 9: Recommending Movies at Scale (Python) 259 Introduction 260 Modeling preference expressions 261 Understanding the data 263 Ingesting the movie review data 266 Finding the highest-scoring movies 270 Improving the movie-rating system 273 Measuring the distance between users in the preference space 276 Computing the correlation between users 280 Finding the best critic for a user 282 Predicting movie ratings for users 285 Collaboratively filtering item by item 288 Building a nonnegative matrix factorization model 292 Loading the entire dataset into the memory 295 Dumping the SVD-based model to the disk 298 Training the SVD-based model 300 Testing the SVD-based model 303 Chapter 10: Harvesting and Geolocating Twitter Data (Python) 307 Introduction 308 Creating a Twitter application 309 Understanding the Twitter API vl.l 312 Determining your Twitter followers and friends 317 Pulling Twitter user profiles 320 Making requests without running afoul of Twitter's rate limits 322 Storing JSON data to the disk 323 Setting up MongoDB for storing Twitter data 325 Storing user profiles in MongoDB using PyMongo 327 Exploring the geographic information available in profiles 330 Plotting geospatlal data in Python 333 Chapter 11: Optimizing Numerical Code with NumPy and SciPv (Python) 339 Introduction 340 Understanding the optimization process 341 Identifying common performance bottlenecks in code 343 pin-

5 Reading through the code 346 Profiling Python code with the Unix time function 349 Profiling Python code using built-in Python functions 350 Profiling Python code using IPython's %tlmelt function 352 Profiling Python code using line_profller 354 Plucking the low-hanging (optimization) fruit 356 Testing the performance benefits of NumPy 359 Rewriting simple functions with NumPy 362 Optimizing the innermost loop with NumPy 366 Index 371 -DD