Python for Data Analysis

Transcription

1 rc3pit tbw Ho 1.-\?;::>_S Python for Data Analysis Wes McKinney OlREILLY Beijing Cambridge Farnham Koln Sebastool Tokyo

2 Table of Contents d i t io n s act our Preface xi arks of related medas re of ~ ssume con- 1. Preliminaries What Is This Book About? 1 Why Python for Data Analysis? 2 ~~~~ 2 Solving the "Two-Language" Problem 2 Why Not Python? 3 Essential Python Libraries 3 NumPy 4 andas 4 matlotlib 5 ~ilioo 5 SciPy 6 Installation and Setu 6 Windows 7 Ale OSX 9 GNU!Linux 10 Python 2 and Python 3 11 Integrated Develoment Environments (IDEs) 11 Community and Conferences 12 Navigating This Book 12 Code Examles 13 Data for Examles 13 Imort Conventions 13 Jargon 13 Acknowledgements Introductory Examles usa.gov data from bit.ly 17 Counting Time Zones in Pure Python 19 iii

3 -- Counting Time Zones with andas MovieLens 1M Data Set Measuring rating disagreement US Baby Names Analyzing Naming Trends Conclusions and The Path Ahead I Python: An Interactive Comuting and Develoment Environment IPython Basics 46 Tab Comletion 47 Introsection 48 The %run Command 49 Executing Code from the Cliboard 50 Keyboard Shortcuts 52 Excetions and Tracebacks 53 Magic Commands 54 Qt-based Rich GUI Console 55 Matlotlib Integration and Pylab Mode 56 Using the Command History Searching and Reusing the Command History Inut and Outut Variables Logging the Inut and Outut 59 Interacting with the Oerating System 60 Shell Commands and Aliases 60 Directory Bookmark System Software Develoment Tools Interactive Debugger Timing Code: %time and %timeit 67 Basic Profiling: %run and %run - 68 Profiling a Function Line-by-Line 70 IPython HTML Notebook 72 Tis for Productive Code Develoment Using!Python 72 Reloading Module Deendencies 74 Code Design Tis 74 Advanced!Python Features 76 Making Your Own Classes!Python-friendly 76 Profiles and Configuration 77 Credits NumPy Basics: Arrays and Vectorized Comutation The NumPy ndarray: A Multidimensional Array Object 80 Creating ndarrays 81 Data Tyes for ndarrays 83 iv I Table otcontents

4 Oerations between Arrays and Scalars Basic Indexing and Slicing Boolean Indexing Fancy Indexing Transosing Arrays and Swaing Axes Universal Functions: Fast Element-wise Array Functions Data Processing Using Arrays Exressing Conditional Logic as Array Oerations Mathematical and Statistical Methods Methods for Boolean Arrays Sorting Unique and Other Set Logic File Inut and Outut with Arrays Storing Arrays on Disk in Binary Format Saving and Loading Text Files Linear Algebra Random Number Generation Examle: Random Walks Simulating Many Random Walks at Once Getting Started with andas Introduction to andas Data Structures 112 Series 112 DataFrame Index Objects Essential Functionality Reindexing Droing entries from an axis Indexing, selection, and filtering Arithmetic and data alignment Function alication and maing Sorting and ranking Axis indexes with dulicate values Summarizing and Comuting Descritive Statistics Correlation and Covariance Unique Values, Value Counts, and Membershi Handling Missing Data Filtering Out Missing Data Filling in Missing Data Hierarchical Indexing Reordering and Sorting Levels Summary Statistics by Level Using a DataFrame's Columns Table of Contents I v

5 Other andas Toics Integer Indexing 152 A Panel Data 6. Data Loading, Storage, and File Formats Reading and Writing Data in Text Format 155 Reading Text Files in Pieces 160 Writing Data Out to Text Format 1 Manually Working with Delimited Formats 163 ]SON Data 165 XML and HTML: Web Scraing 166 Binary Data Formats 171 Using HDF5 Format 171 Reading Microsoft Excel Files 172 Interacting with HTML and Web APis 173 Interacting with Databases 174 Storing and Loading Data in MongoDB Data Wrangling: Clean, Transform, Merge, Reshae Combining and Merging Data Sets 177 Database-style DataFrame Merges D Merging on Index 182 c Concatenating Along an ~"Xis 185 Combining Data with Overla 188 Reshaing and Pivoting 189 Reshaing with Hierarchical Indexing 190 Pivoting "long" to "wide" Format 192 Data Transformation 194 Ii Removing Dulicates 194 Transforming Data Using a Function or Maing 195 Relacing Values 196 (j Renaming ~"Xis Indexes 197 Discretization and Binning 199 Detecting and Filtering Outliers 201 Permutation and Random Samling 202 Comuting Indicator/Dummy Variables 203 String Maniulation 205 String Object Methods 206 I Regular exressions 207 Vectorized string functions in andas 210 I Examle: USDA Food Database 212 vi I Table of Contents

6 Plotting and Visualization A Brief matlotlib API Primer 219 Figures and Sublots 220 Colors, Markers, and Line Styles 224 Ticks, Labels, and Legends 225 Annotations and Drawing on a Sublot 228 Saving Plots to File 231 matlotlib Configuration 231 Plotting Functions in andas 232 Line Plots 232 Bar Plots 235 Histograms and Density Plots 238 Scatter Plots 239 Plotting Mas: Visualizing Haiti Earthquake Crisis Data 241 Python Visualization Tool Ecosystem 247 Chaco 248 maya vi 248 Other Packages 249 The Future of Visualization Tools? Data Aggregation and Grou Oerations GrouBy Mechanics 252 Iterating Over Grous 255 Selecting a Column or Subset of Columns 256 Grouing with Diets and Series 257 Grouing with Functions 2 Grouing by Index Levels 259 Data Aggregation 259 Column-wise and Multile Function Alication 2 Returning Aggregated Data in "unindexed" Form 264 Grou-wise Oerations and Transformations 264 Aly: General slit-aly-combine 266 Quantile and Bucket Analysis 268 Examle: Filling Missing Values with Grou-secific Values 270 Examle: Random Samling and Permutation 271 Examle: Grou Weighted Average and Correlation 273 Examle: Grou-wise Linear Regression 274 Pivot Tables and Cross-Tabulation 275 Cross-Tabulations: Cross tab 277 Examle: 2012 Federal Election Commission Database 278 Donation Statistics by Occuation and Emloyer 280 Bucketing Donation Amounts 283 Donation Statistics by State 285 Table ofcontents I vii

7 R 10. Time Series Date and Time Data Tyes and Tools 290 Converting between string and datetime Time Series Basics 293 Indexing, Selection, Subsetting 294 Time Series with Dulicate Indices 296 Date Ranges, Frequencies, and Shifting 297 Generating Date Ranges 298 Frequencies and Date Offsets 299 Shifting (Leading and Lagging) Data 301 Time Zone Handling 303 Localization and Conversion 304 Oerations with Time Zone-aware Timestam Objects 305 I Oerations between Different Time Zones 306 Periods and Period Arithmetic 307 Period Frequency Conversion 308 Quarterly Period Frequencies 309 Converting Timestams to Periods (and Back) 311 s Creating a Periodindex from Arrays 312 Resamling and Frequency Conversion 312 Downsamling 314 Usamling and Interolation 316 Resamling with Periods 318 Time Series Plotting 319 Moving Window Functions 320 Exonentially-weighted functions 324 Binary Moving Window Functions 324 User-Defined Moving Window Functions 326 Performance and Memory Usage Notes Financial and Economic Data Alications Data Munging Toics 329 Time Series and Cross-Section Alignment 330 Oerations with Time Series of Different Frequencies 332 Aendi Time of Day and "as of" Data Selection 334 Slicing Together Data Sources 336 Index.. Return Indexes and Cumulative Returns 338 Grou Transforms and Analysis 340 Grou Factor Exosures 342 Decile and Quartile Analysis 343 More Examle Alications 345 Signal Frontier Analysis 345 Future Contract Rolling 347 viii I Table ofcontents

8 Rolling Correlation and Linear Regression Advanced NumPy ndarray Object Internals 353 NumPy dtye Hierarchy 354 Advanced Array Maniulation 355 Reshaing Arrays 355 C versus Fortran Order 356 Concatenating and Slitting Arrays Reeating Elements: Tile and Reeat Fancy Indexing Equivalents: Take and Put Broadcasting Broadcasting Over Other Axes Setting Array Values by Broadcasting Advanced ufunc Usage ufunc Instance Methods Custom ufuncs Structured and Record Arrays Nested dtyes and Multidimensional Fields Why Use Structured Arrays? Structured Array Maniulations: numy.lib.recfunctions More About Sorting Indirect Sorts: argsort and lexsort Alternate Sort Algorithms numy.searchsorted: Finding elements in a Sorted Array NumPy Matrix Class Advanced Array Inut and Outut Memory-maed Files HDF5 and Other Array Storage Otions Performance Tis The Imortance of Contiguous Memory Other Seed Otions: Cython, f2y, C Aendix: Python language Essentials Index Table of Contents I ix