Practical Numpy and Matplotlib Intro

Transcription

1 Practical Numpy and Matplotlib Intro presented by Tom Adelman, Sept 28, 2012 What is Numpy? Advantages of using Numpy In [ ]: # an example n = # Python a0 = [i for i in range(n)] time: memory: a1 = [a0[i]+1 for i in range(n)] time: memory: # Numpy b0 = np.arange(n) n= time: memory: b1 = b0 + 1 time: memory: less memory for the same amount of data (about 3-4x less for ints -- 4 for type pointer, 4 for reference count, 4 for value) faster (5-100x) convenient and expressive (e.g., x+1, and much more) functionality (random numbers, math, fancy slicing and indexing, convolutions, statistics, linear algebra, FFTs, etc, etc) Structure of a numpy array in memory In [23]: a0 = array([1, 2, 3, 4, 5], dtype=uint16) "a0:" a0, type(a0) repr(str(a0.data)) a0: [ ] <type 'numpy.ndarray'> '\x01\x00\x02\x00\x03\x00\x04\x00\x05\x00' In [153]: from IPython.core.display import Image as ipimage # just for displaying image in the notebook ipimage(filename='ndarray2.png') Out[153]: Implication: vectorize 1 of 17

2 Summary That is: Numpy = **Multidimensional Homogeneous Arrays** and tools to work with these arrays. Characteristics: 1) all the same type 2) fixed size 3) multidimensional: 1, 2, 3, 4,... 4) x + 1. means add 1 to each element of the array Gain a lot for these restrictions: speed, memory, functionality. Uses examples of uses: Astronomy Artificial intelligence & machine learning Bayesian Statistics Biology (including Neuroscience) Dynamical systems Economics and Econometrics Electromagnetics and Electrical Engineering Geosciences Molecular modeling Signal processing Symbolic math, number theory, etc. Why discuss Numpy (number crunching) and Matplotlib (data plotting together)? In [31]: data = [-.56,.39,.82, 1.47, 2.1] In [32]: plot(data, 'o') xlim(-.5, 4.5)... Always plot your numbers!!! matplotlib gallery page Numpy Basics running numpy ipython 2 of 17

3 In [22]: # to import numpy, do: from numpy import * # or do: import numpy as np # import matplotlib like this... import matplotlib.pyplot as plt # y = exp(sin(x)) # y = np.exp(np.sin(x)) creating ndarrays In [31]: array([1, 5, 10]) # not array(1, 5, 10) [1, 5, 10] [ ] [1, 5, 10] In [15]: zeros((2, 2)) # also ones [[ 0. 0.] [ 0. 0.]] In [25]: arange(10) [ ] In [44]: linspace(-pi, pi, 10) [ ] loadtxt, fromfile, fromstring,.npz files,... can easily read data from: csv, hdf5, R, Matlab, PIL, Excel files,... more basics... In [24]: x = ones((2,2,2), dtype=uint8) # dtype = data type [[[1 1] [1 1]] [[1 1] [1 1]]] In [25]: x[0, 0, 0] = 300 [[[44 1] [ 1 1]] [[ 1 1] [ 1 1]]] # two points: index using [], setting values doesn't conver type ndarray properties 3 of 17

4 In [7]: x = zeros((2,2), dtype=int32) In [8]:.shape (2, 2) In [9]:.dtype int32 basic math: +, -, *, /, ** operations are element-wise In [77]: # degrees Celsius c = linspace(0, 100., 10) c [ ] In [80]: f = (9./5)*c + 32 f [ ] In [30]: x = reshape(arange(4), (2, 2)) **2 [[0 1] [2 3]] [[0 1] [4 9]] In [84]: x = arange(10) y = arange(10) *y 10*x-y [ ] [ ] slicing and indexing [start:stop:step] method In [147]: x = arange(10) [2::2] [::-1] # indexing starts at zero # -1 step reverses direction [ ] [ ] [ ] 4 of 17

5 In [3]: x = arange(10, dtype=uint8) y = x[0:6:2] # slices are views, NOT copies y[2] = 100 y. array_interface ['data'][0], repr(str(x.data)) y. array_interface ['data'][0]#, repr(str(y.data)) [ ] [ ] '\x00\x01\x02\x03d\x05\x06\x07\x08\t' In [4]: z = reshape(x, (2,5)) # slicing and reshaping (if possible), just changes the header, not the z z. array_interface ['data'][0], repr(str(z.data)) [[ ] [ ]] '\x00\x01\x02\x03d\x05\x06\x07\x08\t' To sort out questions during the presentation, the rule of thumb is this: If the function changes the data in the array, then it will return a new array. Reshaping and slicing usually don't require changing the data, so they return a view when possible. To check if you have a copy use the base attribute (ie, y.base is x). By the way, many function have the "out" keyword, and this can be used to do the operation in place, if desired, so sin(x, "out"=x) will put the output of the calculation back into x. Mostly though, I use this not for in-place operations, but when trying to speed up code that's run multiple times. That is, outside the loop I create an array, temp = zeros(( ,)), and then in the loop run sin(x, "out"=temp), so I don't need to constantly recreate space for the output of sin(x). integer indexing e.g.: x[[1, 3, 8]] In [1]: x = arange(10)**2 [array([1, 3, 8])] [ ] [ ] boolean indexing e.g.: x[[false True False True True]] In [3]: x = arange(5) b = array([0, 1, 0, 1, 1], dtype=bool) y = x[b] b y [ ] [False True False True True] [1 3 4] In [2]: x = arange(10) [x%3==0] [ ] [ ] 5 of 17

6 In [5]: [x>3] [ ] In [3]: x[~(x%3==0)] Out[3]: array([1, 2, 4, 5, 7, 8]) assignment with slices In [11]: x = arange(10) x[:2] = 100 [ ] In [14]: x = arange(10) x[x%3==0] = 100 [ ] In [66]: # math on slices x = arange(10) x[x%3==0] += 100 [ ] functions (ufuncs and others) There are many, many of these. Ufuncs are the basics, and have a few extra abilities ufuncs examples: add, multiply, sin, exp, log, sqrt, bitwise_and, greater, logical_and, floor also: maximum and minimum (which should generally be used instead of Python's max and min In [107]: Out[107]: x = linspace(0, 10*pi, 10000) y = sin(x) plot(x, y) [<matplotlib.lines.line2d at 0x1886f550>] 6 of 17

7 In [108]: Out[108]: (-1, 1) z = clip(y, -.8,.8) plot(x, z); ylim(-1, 1) In [109]: Out[109]: y[y>=.8] =.5 y[y<=-.8] = -.5 plot(x, y) [<matplotlib.lines.line2d at 0x273d9650>] In [110]: Out[110]: mask = (y<.4)&(y>-.4) x[mask] += 1 plot(x, y) [<matplotlib.lines.line2d at 0x1ef4c0d0>] At this point, you know enough to make a lot of progresss. Think of the operations you want, find them in Numpy and apply them. 7 of 17

8 functions and techniques for shape manipulation (more advanced) In [28]: Out[28]: x = linspace(0, 10*pi, ) y = sin(x) +.1*sin(1000*x) plot(x, y) [<matplotlib.lines.line2d at 0x43be410>] In [26]: from IPython.core.display import Image as ipimage # just for displaying image in the notebook ipimage(filename='min01.png') Out[26]: 8 of 17

9 In [29]: m = 1000 y0 = reshape(y, (-1, m)) # y.shape = (-1, 1000) ymin = minimum.reduce(y0, axis=1) ymax = maximum.reduce(y0, axis=1) ymid = sum(y0, axis=1)/m plot(x[::m], ymin, x[::m], ymax, color='k') fill_between(x[::m], ymin, ymax, color='g') plot(x[::m], ymid, color='r') y.shape, y0.shape, ymin.shape, ymax.shape, ymid.shape ( ,) (1000, 1000) (1000,) (1000,) (1000,) In [12]: # interlude with numpy functions "sum" and "minimum" seed(8) temp0 = randint(0, 9, (2,3)) temp1 = randint(0, 9, (2,3)) temp0 temp1 "sum temp0: " "axis=0" sum(temp0, axis=0) # numpy.sum(a, axis=none, dtype=none, out=none, keepdims=false)" "axis=1" sum(temp0, axis=1) "minimum(temp0, temp1):" minimum(temp0, temp1) # numpy.minimum(x1, x2[, out]) "reduce temp0:" minimum.reduce(temp0, axis=0) minimum.reduce(temp0, axis=1) [[3 4 1] [5 8 3]] [[8 0 5] [1 3 2]] sum temp0: axis=0 [ ] axis=1 [ 8 16] minimum(temp0, temp1): [[3 0 1] [1 3 2]] reduce temp0: [3 4 1] [1 3] 9 of 17

10 In [14]: # "concatenate" # triangle wave (-2 to 3, 200 pts per period) a0 = linspace(-2, 5, 100, endpoint=false) # endpoint=false a1 = linspace(5, -2, 100, endpoint=false) b0 = concatenate((a0, a1)) plot(b0) xlim(-10, 210) ylim(-2.5, 5.5) Out[14]: (-2.5, 5.5) In [80]: Out[80]: b1 = concatenate((a0, a1)*6) plot(b1) [<matplotlib.lines.line2d at 0x59a4210>] 10 of 17

11 In [81]: # "resize" new_length = int(len(b0)*6.2) len(b0), new_length b1 = resize(b0, (new_length,)) plot(b1) Out[81]: [<matplotlib.lines.line2d at 0x59ac9d0>] In [4]: # how about "repeat"? x = arange(5) repeat(x, 4) [ ] [ ] In [33]: # newaxis "x.shape = ", x.shape "x[newaxis,:].shape = ", x[newaxis,:].shape y = repeat(x[newaxis, :], 4, axis=0) y x.shape = (5,) x[newaxis,:].shape = (1, 5) [[ ] [ ] [ ] [ ]] In [69]: # y.flat y.flat array(y.flat) # y.flat is an iterator, array(y.flat) is an ndarray <numpy.flatiter object at 0x > [ ] 11 of 17

12 In [35]: b2 = array( repeat(b0[newaxis, :], 6, axis=0).flat ) plot (b2, 'r') Out[35]: [<matplotlib.lines.line2d at 0x5099b90>] broadcasting In [38]: seed(4) x0 = randint(0, 9, (3,2)) x1 = array([5, 10]) 0 1 y = repeat(x1[newaxis,:], 3, axis=0) y 0*y [[7 5] [1 8] [7 8]] [ 5 10] [[ 5 10] [ 5 10] [ 5 10]] [[35 50] [ 5 80] [35 80]] In [85]: # broadcasting... like an automatic newaxis and repeat (without actually creating the array) y = x0*x1 y [[35 10] [ 0 60] [20 0]] 12 of 17

13 In [36]: x = linspace(-1, 1, 512) f0 = exp(-(x)**2/.05) plot(x, f0) Out[36]: [<matplotlib.lines.line2d at 0xb2a52b0>] In [37]: Out[37]: # newaxis with broadcasting! f02d = f0[newaxis,:]*f0[:,newaxis] imshow(f02d, cmap=cm.gray) <matplotlib.image.axesimage at 0x4d0f470> Numpy plays well with others... PIL 13 of 17

14 In [20]: Out[20]: from IPython.core.display import Image as ipimage # just for displaying image in the notebook ipimage(filename='lena.png') In [1]: import Image as pilimage # PIL's Image class im = pilimage.open("lena.png") im.size (512, 512) In [2]: # convert PIL image to Numpy ndarray a = array(im) a.shape type(a), a.dtype (512, 512, 3) <type 'numpy.ndarray'> uint8 14 of 17

15 In [3]: # convert to black and white using a simple sum of all colors... a = array(a, dtype=uint32) bw = sum(a, axis=2) bw.dtype, minimum.reduce(bw.flat), maximum.reduce(bw.flat) imshow(bw, cmap=cm.gray) Out[3]: uint <matplotlib.image.axesimage at 0x48ad9d0> In [47]: Out[47]: bw1 = bw.copy() bw1[bw1<150] = 1000 imshow(bw1, cmap=cm.gray) <matplotlib.image.axesimage at 0x570b4f0> In [48]: Out[48]: bw1 = roll(bw, 200, axis=1) imshow(bw1, cmap=cm.gray) <matplotlib.image.axesimage at 0x59315f0> 15 of 17

16 In [4]: x = linspace(0, 10*pi, bw.shape[0]) bw2 = bw*(1+sin(x)) imshow(bw2, cmap=cm.gray) Out[4]: <matplotlib.image.axesimage at 0x4a62b90> In [50]: Out[50]: x, y = mgrid[-1:1:512j, -1:1:512j] mask = (x**2 + y**2)<0.6**2 bw3 = bw.copy() bw3[~mask]=0 imshow(bw3, cmap=cm.gray) <matplotlib.image.axesimage at 0x6857d70> In [52]: x, y = mgrid[-2:3,-2:3] y [[ ] [ ] [ ] [ ] [ ]] [[ ] [ ] [ ] [ ] [ ]] ctypes 16 of 17

17 In [ ]: #ndarray.ctypes.data # C header C header C header C header C header C header int32 DAQmxWriteAnalogF64 (TaskHandle taskhandle, int32 numsampsperchan, bool32 autostart, float64 bool32 datalayout, float64 writearray[], int32 *sampsperchanwritten, bool32 *r #Python Python Python Python Python Python nidaq.daqmxwriteanalogf64(taskhandle, int32(numsampsperchan), int32(autostart), float64(timeout), datalayout, writearray.ctypes.data, ctypes.byref(sampsperchanwritten), None)) return sampsperchanwritten.value # that is, numpy_array.ctypes.data returns data that can be used when calling a C function with Demos: spectrum from computer's microphone In [ ]: np_data = fromstring(microphone_data, dtype=np.int16) spect = log10(abs(fft.rfft(np_data))) handwritten number classification using feedforward neural network In [38]: from IPython.core.display import Image as ipimage # just for displaying image in the notebook ipimage(filename='nn.png') Out[38]: In [ ]: X1 = np.concatenate((np.ones((m,1)), X), axis=1) z2 = np.dot(x1,theta1.t) a2 = sigmoid(z2) a2 = np.concatenate((np.ones((a2.shape[0],1)), a2), axis=1) z3 = np.dot(a2, Theta2.T); a3 = sigmoid(z3) ix = np.argmax(a3, axis=1) 17 of 17