Statistical Data Analysis: Python Tutorial

Transcription

1 1 October 4, 2017 Statistical Data Analysis: Python Tutorial Dr A. J. Bevan, Contents 1 Getting started 1 2 Basic calculations 2 3 More advanced calculations 4 4 Data sets CSV file input Plotting data 7 1 Getting started Python is an open source statistical programming language which can be downloaded from the following website: This scripting language is useful for this course, and some sectors value familiarity with Python. You may wish to use these tutorials to introduce yourself to this package to bolster your CV. In these notes type in bold is entered into a command line terminal, and italicised text appears on the command line as a result. We will use Anaconda that is a bundle that comes with python as well as a GUI called Spyder. The latter provides you with the opportunity to write scripts in an integrated environment without needing to know how to use a separate editor. Anaconda can be downloaded from The student services Windows 10 machines at QMUL have anaconda installed, however given the nature of the setup you will need to copy a shortcut to the anaconda install that will be used. The following directory on the J drive J:\Physics\Teaching\spa6328 contains this shortcut (LaunchSpyder-anaconda3) as well as the python example scripts prepared for the module. You should copy the content of this directory to your area (e.g. Desktop) in order to start working with the files. a.j.bevan at qmul.ac.uk

2 2 The shortcut itself is setup to run the following command 1. %windir%\system32\cmd.exe /K L:\Anaconda3\python3.5\Scripts\activate.bat && spyder The first part (up to and including /K ) starts a command terminal, the second part (up to activate.bat) sets the anaconda environment, including the python path, and the last part launches Spyder. 2. The shortcut should start on your desktop. This is specified as G:\Desktop In order to start Python from a LINIX/UNIX or Mac OSX computer that has python or anaconda installed correctly just open a Darwin (Mac) or command line terminal and type the following at the command prompt python This should result in the following output being displayed on the command line terminal Python Anaconda (x86 64) (default, May , 13:04:09) [GCC Compatible Apple LLVM 6.0 (clang )] on darwin Type help, copyright, credits or license for more information. >>> As noted above, the use of the python command line tool is limited and you will soon want to write python scripts, where we resort to Spyder. First open the anaconda-navigator (type this at the terminal prompt and press return). Once the navigator has opened you should see Spyder and a button that when clicked will launch this GUI. In order to quit an interactive Python session one simply enters quit() or Ctrl-D at the prompt (>>>). More information on the use of Python can be found online at the Python website and as the result of performing a google search for Python. In addition to this there are a number of books available on the use of this language. 2 Basic calculations You can perform simple arithmetic in Python in a logical way using operators akin to any other programming language. For example 3* / It is convenient to abstract quantities and refer to them via names. Different data types exist, and above we see integers being used. Python dynamically determines the type of a variable based on the usage of that variable (unlike most other languages). For example to create a variable called x and assign a number to it we can enter x = 3 At some later time we can re-assign a value to that variable, and we can refer to the variable as x without having to worry about what the actual value is assigned to this. Integer division will be evaluated at the command prompt as the division of two real numbers, but in a script will be taken as the division of two integers. This can lead to an incorrect computation, so be careful to ensure that you have the correct calculation being performed.

3 3 There is a command line help function that enables you to find out more about a particular built-in Python command. This will not work for classes that are in modules (see below for modules). For example if you wish to query the help for the built in type list simply enter help(list) at the Python prompt. This will result in the following output. Help on class list in module builtins: class list(object) list() -> new empty list list(iterable) -> new list initialized from iterable s items Methods defined here: add (self, value, /) Return self+value.... The real power of a language is the higher level functionality provided by libraries. Basic mathematical operations are built in types in Python. For more advanced computation (for example trig functions) we need to import the math library. This is done by typing import math The functions provided by this library can be seen by first importing the library and then help(math) The following functions are available in the math library : acos inverse cosine acosh inverse hyperbolic cosine asin inverse sine asinh inverse hyperbolic sine atan/atan2 inverse tangent atanh inverse hyperbolic ceil round up argument to nearest integer cos cosine cosh hyperbolic cosine degrees convert from radians to degrees erf error function erfc complementary error function exp exponential expm1 exponential minus 1 (use for precision when computing exp(x)-1) fabs absolute value of a float factorial factorial floor round down to the nearest integer gamma gamma function isnan return TRUE if x is not a number, otherwise FALSE log natural logarithm log10 log base 10 pow raise x to some power radians convert degrees to radians sin sine sinh hyperbolic sine sqrt square root tan tangent tanh hyperbolic tangent Note that trig functions are computed in radians. Not all functions in math are listed here.

4 4 Note that function names in Python are followed by arguments in parentheses. So for some number x, one can compute the square root of x as sqrt(x) and so on. In addition to this one can define arrays of numbers and perform simple calculations on these sets of data. The following example creates an ordered list of data in an array with variable name x, and computes the arithmetic mean, variance and standard deviation for these data. AnArray = [1, 2, 3, 4] print(anarray) [1, 2, 3, 4] numpy.mean(anarray) 2.5 numpy.var(anarray) numpy.std(anarray, ddof=1) NumPy has many other functions, including average ceil cov floor median prod sum Compute the weighted average Round up to the nearest integer Estimate the covariance matrix Round down to the nearest integer Determine the median Return the product Compute the sum The user guide for this package can be found at Exercises 1. Define the variables θ = 3.2 and φ = 1.7 and print out these variables. 2. Compute θ + φ. 3. Compute θ φ. 4. Compute θφ. 5. Compute θ/φ. 6. Define an array x containing the prime numbers between 1 and Print out the array x. 8. Compute the arithmetic mean, variance and standard deviation of x. 3 More advanced calculations It is possible to perform more advanced calculations using additional functionality in Python, for example via numpy or the math libraries. However sometimes we need to define higher level functionality in a script

5 5 that is not already implemented. For this we need to utilise more advanced control features such as loop statements and functions. Loops: allow us to iterate over variables like arrays. For example the numpy function mean for some array can be coded up in python as a loop in the following way AnArray = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] sum = 0 for i in AnArray: sum += AnArray[i-1] sum = sum/len(anarray) Note that python determines how many elements are in the array, and when indexing the array, the number of elements starts being counted at zero (0) and runs to N-1. This convention is the same as other popular languages (e.g. C, C++, Java), but differs from older languages such as FORTRAN (still used for some scientific programming). The first line defines the data array to be analysed; the second provides an initilised counter variable; the third is a loop over elements in the variable AnArray; within this loop there is an indentation (this is how Python deals with scope); the last line is normalising the sum over elements to compute the average. In terms of pseudo-code we can express this calculation as: 1. Define the data 2. Sum over the data 3. Divide the data by the number of elements Each time you have to perform a complicated operation, such as the arithmetic average calculation above, it would be unproductive to have to re-write the code. We can turn this code snippet into a re-usable element by using the concept of a function. Let s call this function average, and we can define the function average as def average(a): sum = 0 for i in a: sum += a[i-1] sum = sum/len(a) return sum Now when we call this function av = average(anarray) av will be assigned the value corresponding to the average. This will be 5.5 using the data provided above. Exercises 1. Using the data set Ω = {1, 1.2, 1.5, 1.7, 2.3} compute the arithmetic mean without resorting to the built in function. 2. Using the data set Ω = {1, 1.2, 1.5, 1.7, 2.3} compute the standard deviation without resorting to the built in function. 3. Using the data set Ω = {1, 1.2, 1.5, 1.7, 2.3} compute the skew without resorting to the built in function. 4 Data sets There are several data sets that are available via SciKitLearn. To access these data sets you will need that package installed, then you can load datasets from sklearn into your session via from sklearn import datasets

6 6 The sda 2.py example script will use the Iris data from the Fisher publication, where the aim of that paper was to classify the different types of Iris using features of the data. The paper itself will be a reading homework later in the module. The Iris data includes four features: Sepal length (cm) Sepal width (cm) Petal length (cm) Petal width (cm) These are extracted from the dataset as X using the commands below iris = datasets.load iris() X = iris.data[:, :4] Y = iris.target Y is the target variable (the type of iris: either setosa, versicolor or verginica). It is possible to divide the ensemble of data up into the individual types of iris using Xsetosa = X[Y==0] Ysetosa = Y[Y==0] Xversicolor = X[Y==1] Yversicolor = Y[Y==1] Xvirginica = X[Y==2] Yvirginica = Y[Y==2] This helps with plotting the features for one particular type of the data. SciKit Learn has several other example data sets that are described at CSV file input Often we are interested in analysing data collated in a CSV file format (e.g. from an Excel spread sheet or a flat file). Some HEP data (for example the data format provided for the Kaggle Higgs data challenge) is represented as a CSV file with real valued and string valued data elements. As a result there is an example CSV file reader that addresses the issues with extracting data into columns that are numerical and columns that are string data. If only one type of data exist then the arrays for the other type is returned null. The file CSV IO.py includes a function ReadCSV that takes one or two arguments; the first argument being the CSV filename and the second being either quiet or verbose (controls the level of information printed out when reading the file). The following shows an example of how to use this functionality import CSV IO.py dat, col, strdat, strcol = CSV IO.ReadCSV( test data.csv ) Here dat and strdat are the numerical and string data, respectively. The corresponding column names are stored in col and strcol. The example data file test data.csv only has numerical data with 6 entries and three columns (x, y and z). Exercises

7 7 1. Inspect (printout) the Fisher data set to understand how the arrays are related and to make sure you are comfortable with selecting sub-sets of an array by cutting on values. 2. Extract one of the other data types (e.g. Boston house prices) and try to understand the format by comparing with the Fisher data. 5 Plotting data matplotlib is used in sda 2.py to illustrate plotting in python. There are histogram and scatter plot examples, where the Fisher data described above are used to make plots. The starting point is to import the matplotlib.pyploy functionality using import matplotlib.pyplot as plt Having split the Fisher data up into different types of iris, we can choose to plot the first feature of the iris data set for type setosa using the following fig hist = plt.figure() ax = fig hist.add subplot(1, 1, 1) ax.hist(xsetosa[:, 0], 10, label= setosa ) plt.title( Iris Setosa ) ax.set xlabel( Sepal length (cm) ) ax.set ylabel( Number of entries ) plt.show() This will produce a plot like the one shown in Figure 1. Figure 1: The sepal length data for iris setosa from the Fisher data sample. Sometimes we want to visually inspect the change in one variable as a function of another. An example of this is shown in Figure 2, where the iris sepal length is plotted against the sepal width for the three iris types. The corresponding python to create this plot is

8 8 plt.figure(2, figsize=(8, 6)) plt.scatter(xsetosa[:, 0], Xsetosa[:, 1], c= b, cmap=plt.cm.set1, edgecolor= k, label= setosa ) plt.scatter(xversicolor[:, 0], Xversicolor[:, 1], c= r, cmap=plt.cm.set1, edgecolor= k, label= versicolor ) plt.scatter(xvirginica[:, 0], Xvirginica[:, 1], c= g, cmap=plt.cm.set1, edgecolor= k, label= verginica ) # The plot needs to have axis labels and the legend should be appropriately placed plt.legend(loc= upper left ) plt.xlabel( Sepal length (cm) ) plt.ylabel( Sepal width (cm) ) # set appropriate plot ranges to avoid having a crowded image x min, x max = X[:, 0].min() -.5, X[:, 0].max() +.5 y min, y max = X[:, 1].min() -.5, X[:, 1].max() +.5 plt.xlim(x min, x max) plt.ylim(y min, y max) plt.show() Figure 2: The sepal length and width data for different iris types found in the Fisher data sample. Exercises Adapt the sda 2.py script to do the following 1. Create a histogram of the petal length for each type of iris using the Fisher data; Add axis labels to the plot you ve created. 2. Save the plot to a file (click on the disc icon to open the save dialogue window). 3. Create a scatter plot of the petal length vs petal width for each type of iris using the Fisher data. As before include axis labels, and now also include a legend. More information (including extensive source of documentation) can be found on the following web pages: Python: Anaconda: NumPy: Matplotlib: