Command Line and Python Introduction. Jennifer Helsby, Eric Potash Computation for Public Policy Lecture 2: January 7, 2016

Transcription

1 Command Line and Python Introduction Jennifer Helsby, Eric Potash Computation for Public Policy Lecture 2: January 7, 2016

2 Today Assignment #1! Computer architecture Basic command line skills Python fundamentals Variables and types Basic data structures First programs Basic plotting with Python s matplotlib

3 Assignment #1 Goal: Get familiar with Python. Learn to do simple Python programming and make simple plots Use crime statistics data from the City of Chicago data portal Due in 1 week: January 14, 2016 Submit via to the course account

4

5 Class Virtual Machine Class VM download: Link on Chalk Load with VirtualBox (available for Mac, Windows, GNU/Linux) Feel free to install software yourself

6 Hardware RAM HDD CPU I/O

7 Software user 1 user 2 user 3... user n system and application programs operating system hardware Adapted from Silberschatz, Galvin and Gagne

8 Operating System Does things that the user doesn t need/want to deal with Makes the system more efficient and convenient to use Intermediary between the hardware and user

9 Unix and its variants Unix is family of extremely popular operating systems Variants: GNU/Linux Mac OS X BSD Unix

10

11 Command Line: Mac OS X

12 Command Line: Debian GNU/Linux

13 Command Line Syntax

14 Command Line: Basic Unix Commands 1 : List contents of directory : Print current working directory : Change directory : Make new directory : Copy a file or directory (with ) : print to screen (really concatenate)

15 Command Line: Basic Unix Commands 2 : Remove file : Remove directory : Text editor in terminal : Get file from the web Use rm with caution!!

16 Navigating Files Look at first few or last few lines using and : (especially important for very large files) Search through files looking for certain strings using :

17

18 Notation Default directory is a user s home directory, e.g. on the VM Current directory Parent directory

19 Notation Default directory is a user s home directory, e.g. on the VM Current directory Parent directory What directory is?

20 Redirecting Output Send output of as input to Send output of to a new file Append output of to an existing file

21 Example: Redirection Download the current employee names, salaries, and position titles of city employeesfrom the City of Chicago data portal ( ) Let s count the number of police officers in the dataset using grep and redirection: counts the number of lines Can also redirect to and page through the results:

22

23 Example: Redirection Take only the detectives and redirect the output to a new text file:

24

25 Python Popular and easy to learn interpreted language Two versions of Python commonly used: 2.7x - Run with and 3.x (recommended) - Run with and Running Python code: From command line From interpreter

26 Python: Interpreter Quit Python with or CTRL-D

27 Python: Command Line

28 Python: Basic Data Types Integer: String: List: Floating point: Boolean:, Dictionary: or or

29 Python: Variables Assign values to variables using the assignment operator (=):

30 Python: Arithmetic Simple integer arithmetic: Arithmetic operators: (discard remainder) ** (raise to the power) (mod, return remainder of division),

31 Python: String Manipulation Assign a string using : Indexing a string using square brackets: index dog_name R o v e r

32 Python: String Manipulation Concatenate (join strings together) with and repeat them N times with :

33 Python: String Methods 1 Split a string on spaces using Split on another character, :, using :

34 Python: String Methods 2 Return the uppercase and lowercase versions of a string with : Replace a substring with another using : and

35 Python: Lists Assign lists with comma-separated values in square brackets: Index with square brackets: index favorite_numbers

36 Python: Length of list or string Determine length of list with: Similarly with strings:

37 Python: List Methods 1 Append a new value with or multiple values with : index favorite_numbers And then pop them off with :

38 Python: List Methods 2 Reverse list with Sort lists with : :

39 Python: Dictionaries Dictionaries are key value stores Define using: Reference a given entry using square brackets

40 Python: Boolean Comparison operators: (not equal to) Combine with and (less than or equal to), :, (equal to),,,

41 Python: Data Types Determine data types using :

42 Python: Loading Python modules Wide array of modules in Python available for almost any computational task Use existing code in these libraries when you can to prevent reinventing the wheel Libraries are imported into the Python environment using the statement:

43 Other Python Tidbits Whitespace (tabs, spaces) is meaningful Will talk more about this next time Comment with

44 Important Python Packages for Scientific Computing Scipy: Numpy: Library for fast computations on large arrays pandas: Library of codes for scientific computing (linear algebra, stats, interpolation,...) Popular data analysis toolkit matplotlib Most popular package for plotting

45 Numpy ndarray : n-dimensional array Useful for linear algebra and image manipulation

46 Numpy: Useful Functions : Makes linearly spaced bins Generate random numbers with Make specific matrices with : and :

47 IPython Notebook Useful way of doing data analysis (collaborative) Launch with: Can do Python programming in browser environment

48

49 IPython: Integrating plotting Enable plotting inside the notebook with:

50 Pandas Extremely useful data analysis package perfect for analysis in social science, policy, etc. Two fundamental objects: DataFrame: 2D object Series: 1D object Nicely handles missing data, alignment of data, merging and joining datasets, handling timeseries data

51 Pandas: Basic Data Handling Read in a CSV file into a Pandas DataFrame: Pandas DataFrame:

52 Pandas: Columns (Series) Refer to a single column using square brackets:

53 Pandas: Munging Data Need floating point numbers only so that we can make a histogram of salaries Problem 1: NaNs Problem 2: Strings

54 Pandas: Munging Data Solution 1: Use to drop rows with NaNs: Solution 2: Strip off dollar signs with point with : We can now visualize this data! and represent as floating

55 Matplotlib Basic Plotting: Histogram Histogram with :

56 Matplotlib Basic Plotting: Histogram Histogram with :

57 Pandas plotting with : Bar Chart

58 Pandas Supported Plot Types Pass string to to tell pandas which plot to make Supported plot types you will find useful: Bar charts: bar or barh Histograms: hist Scatter plots: scatter Pie charts: pie